PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison

摘要

The selection, development, or comparison of machine learning methods in datamining can be a difficult task based on the target problem and goals of aparticular study. Numerous publicly available real-world and simulatedbenchmark datasets have emerged from different sources, but their organizationand adoption as standards have been inconsistent. As such, selecting andcurating specific benchmarks remains an unnecessary burden on machine learningpractitioners and data scientists. The present study introduces an accessible,curated, and developing public benchmark resource to facilitate identificationof the strengths and weaknesses of different machine learning methodologies. Wecompare meta-features among the current set of benchmark datasets in thisresource to characterize the diversity of available data. Finally, we apply anumber of established machine learning methods to the entire benchmark suiteand analyze how datasets and algorithms cluster in terms of performance. Thiswork is an important first step towards understanding the limitations ofpopular benchmarking suites and developing a resource that connects existingbenchmarking standards to more diverse and efficient standards in the future.